Update to cime5.3-alpha10 #1490
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Now --user-mods on the command line (including testmods) will take
precedence over the user_mods set by the compset - for user_nl files,
shell_commands and SourceMods.
I have tested this with this diff to the A compset
diff --git a/src/drivers/mct/cime_config/config_compsets.xml b/src/drivers/mct/cime_config/config_compsets.xml
index c11354e..7e6c2c9 100644
--- a/src/drivers/mct/cime_config/config_compsets.xml
+++ b/src/drivers/mct/cime_config/config_compsets.xml
@@ -40,6 +40,7 @@
<compset>
<alias>A</alias>
<lname>2000_DATM%NYF_SLND_DICE%SSMI_DOCN%DOM_DROF%NYF_SGLC_SWAV</lname>
+ <user_mods>/Users/sacks/temporary/user_mods_compset</user_mods>
</compset>
<compset>
Along with this create_newcase command:
./create_newcase -case test_0414m -compset A -res f45_g37 \
--run-unsupported \
--user-mods-dir /Users/sacks/temporary/user_mods_command_line
where the contents of the two relevant user_mods directories are:
--- user_mods_compset/shell_commands ---
./xmlchange STOP_N=101
--- user_mods_compset/SourceMods/src.drv/mysrc.F90 ---
user_mods_compset
--- user_mods_compset/user_nl_cpl ---
user_mods_compset
--- user_mods_command_line/shell_commands ---
./xmlchange STOP_N=102
--- user_mods_command_line/SourceMods/src.drv/mysrc.F90 ---
user_mods_command_line
--- user_mods_command_line/user_nl_cpl ---
user_mods_command_line
The final contents are:
--- user_nl_cpl ---
user_mods_compset
user_mods_command_line
--- shell_commands ---
./xmlchange --force STOP_N=102
--- SourceMods/src.drv/mysrc.F90 ---
user_mods_command_line
And
$ ./xmlquery STOP_N
STOP_N: 102
thus demonstrating that the user_mods on the command-line takes
precedence over the compset's user_mods.
This was already the policy, but was not being enforced correctly.
The previous implementation had two problems: 1. If you specified a user_mods on the command-line along with a compset that has its own user_mods, then the compset's user_mods get applied twice. 2. The new place where there was a call to apply_user_mods happened too early: xmlchange commands can not be done at that point. This fixes these problems. I have tested this with the same changes described in 9cc7740. I tested create_newcase with no user_mods, user_mods just from the command line, user_mods just from the compset, and user_mods from the command line and the compset.
Fixes for PRE test
Remove conditional pio1 code
Make Machines.get_value more likely to return values of the correct type
Inform user of machine
Force user to always go through case.submit
Big cs.status upgrade
Now --user-mods on the command line (including testmods) will take
precedence over the user_mods set by the compset - for user_nl files,
shell_commands and SourceMods.
I have tested this with this diff to the A compset
diff --git a/src/drivers/mct/cime_config/config_compsets.xml b/src/drivers/mct/cime_config/config_compsets.xml
index c11354e..7e6c2c9 100644
--- a/src/drivers/mct/cime_config/config_compsets.xml
+++ b/src/drivers/mct/cime_config/config_compsets.xml
@@ -40,6 +40,7 @@
<compset>
<alias>A</alias>
<lname>2000_DATM%NYF_SLND_DICE%SSMI_DOCN%DOM_DROF%NYF_SGLC_SWAV</lname>
+ <user_mods>/Users/sacks/temporary/user_mods_compset</user_mods>
</compset>
<compset>
Along with this create_newcase command:
./create_newcase -case test_0414m -compset A -res f45_g37 \
--run-unsupported \
--user-mods-dir /Users/sacks/temporary/user_mods_command_line
where the contents of the two relevant user_mods directories are:
--- user_mods_compset/shell_commands ---
./xmlchange STOP_N=101
--- user_mods_compset/SourceMods/src.drv/mysrc.F90 ---
user_mods_compset
--- user_mods_compset/user_nl_cpl ---
user_mods_compset
--- user_mods_command_line/shell_commands ---
./xmlchange STOP_N=102
--- user_mods_command_line/SourceMods/src.drv/mysrc.F90 ---
user_mods_command_line
--- user_mods_command_line/user_nl_cpl ---
user_mods_command_line
The final contents are:
--- user_nl_cpl ---
user_mods_compset
user_mods_command_line
--- shell_commands ---
./xmlchange --force STOP_N=102
--- SourceMods/src.drv/mysrc.F90 ---
user_mods_command_line
And
$ ./xmlquery STOP_N
STOP_N: 102
thus demonstrating that the user_mods on the command-line takes
precedence over the compset's user_mods.
The previous implementation had two problems: 1. If you specified a user_mods on the command-line along with a compset that has its own user_mods, then the compset's user_mods get applied twice. 2. The new place where there was a call to apply_user_mods happened too early: xmlchange commands can not be done at that point. This fixes these problems. I have tested this with the same changes described in 9cc7740. I tested create_newcase with no user_mods, user_mods just from the command line, user_mods just from the compset, and user_mods from the command line and the compset.
Point is: I want to make it hard to miss
…into mvertens/usermods_in_compset
jgfouca
approved these changes
May 9, 2017
|
I'd like to do a test merge to master and look at it with gitx to make sure it doesn't add another instance of the complete CIME history, |
rljacob
approved these changes
May 14, 2017
|
Is this ready to go? |
Upstream merge to resolve conflicts. * master: (93 commits) Adds the --force-move option and implies --copy-only when --last-date is specified without --force-move added spunup A_WCYCL options (1850S and 2000S) for v0atm Updates land initial conditions for ne120_oRRS18v3 Fix jenkins_generic_job when user selects specific machines. Fix small problems with MPAS components from debug tests Adds a warning when using the --last-date option and to its help Implement the copy_only option for short term archiving. This copies files rather than moving them Implemented most of the machinery for testing with "incomplete" log files Fix code format issue - replace unused variable with _ Update template.st_archive Adds options to the st_archive to specify the last date (--last-date) to archive, and whether to disable archiving incomplete log files (--no-incomplete-logs) Turn off salinity restoring by default until it passes exact restart This is a fix to a PR #1501 (github issue #1263). I allocated the tmp_array to be too large and GNU compiler stopped with an array mismatch. Change this to be state%ncol in size, not pcol Fix a floating invalid that we found with certain PE's (and with intel v17.02) bug fix Revert mpas-o commit with new threaded vector reconstruction consistency changes Changing jobmin to 1 for batch q on blues Modifies SMS_D_Ln5_P8x4 to SMS_Ln5 to avoid cetus and melvin issues Point at corrected salinity restoring files for oEC60to30v3 and oRRS18to6v3 Make sure clean clm cleans up obj dir ...
jgfouca
added a commit
that referenced
this pull request
May 15, 2017
Update to cime5.3.1 Pulling in another 2 weeks of CIME changes into ACME. CIME hash 15297cd from May 2. Required bug fixes for: Changing error to warning for using "-id" with single dash acme templates missing "arg =" Restore pio1 Cmake path pointing to pio2/cmake Requires #1464 to be pushed and merged Update to gcc5.3.0 to avoid internal compiler error Still had 2 fails of acme_developer on penn machine, but look to be weird gcc issues -- var tracking overflow. Same tests passed on redsky, so expecting it will pass on ACME machines. * agsalin/update-to-cime5.3.1: (216 commits) Make single-dash before multichar arg a warning Fix acme template change for new parser Update Sandia worksations to gcc5.3.0 Comment out invalid_args check Revert change in pio1 to point into pio2/cmake Fix single submit Add tests Do not override walltime unless test Update ChangeLog nag compiler needs a width Add checks to verify the create_newcase directory was created as expected removed unpack commented region fix typo in parse_args Bug fixes Minor pylint fix fix pylint issues rename function updated config_grids.xml to be the same as master fixed comment on new aquaplanet mode updates to have aquaplanet not depend on a new grid definition with a null mask ...
|
OK, it's on next. |
|
Why did you merge master to this branch? |
|
@rljacob To resolve conflicts |
|
Summary of testing on next: Many expected namelist diffs: Several pio namelist vars: @jayeshkrishna said: ESMCI/cime#1441 was an NML change that reverted a feature that reset strides/tasks etc by default - from "-99" to a value . This caused other issues as discussed in ESMCI/cime#1433 and the resolution was to remove the feature. So the values of strides/num io tasks etc should now be -99 etc, the defaults that were set before the feature was introduced in drof_in There were several tests that would have reported only namelist diffs but had FAIL because of a problem with the memory test. This affected any test which had not been blessed in a while and so had an old-format cpl.log Fixed by @jgfouca and re-merged to next. The ERP test was changed to use BUILD_THREADED=TRUE in all builds which caused some baseline diffs in these tests because they have nthrds=1 one and so used to be built with BUILD_THREADED=FALSE: These tests had the mem test fail, namelist diffs AND baseline diffs for reasons we are still trying to understand: |
|
@jgfouca I'm trying to run a test on blues with this branch and getting an xmllint error: xmllint version: |
|
@jgfouca the PR description also needs a list of ACME bugs fixed. Issues that were "fixed in CIME". |
|
@rljacob done |
jgfouca
added a commit
that referenced
this pull request
May 18, 2017
Update to cime5.3-alpha10 Pulling in another 2 weeks of CIME changes into ACME. CIME hash 15297cd from May 2. Required bug fixes for: Changing error to warning for using "-id" with single dash acme templates missing "arg =" Restore pio1 Cmake path pointing to pio2/cmake Update to gcc5.3.0 to avoid internal compiler error Still had 2 fails of acme_developer on penn machine, but look to be weird gcc issues -- var tracking overflow. Same tests passed on redsky, so expecting it will pass on ACME machines. [BFB] * agsalin/update-to-cime5.3.1: (219 commits) Add fix to bug in docn that was in this cime version Fix upstream merge resolution mistake Bug fix: Handle failures to get mem usage from baselines Make single-dash before multichar arg a warning Fix acme template change for new parser Update Sandia worksations to gcc5.3.0 Comment out invalid_args check Revert change in pio1 to point into pio2/cmake Fix single submit Add tests Do not override walltime unless test Update ChangeLog nag compiler needs a width Add checks to verify the create_newcase directory was created as expected removed unpack commented region fix typo in parse_args Bug fixes Minor pylint fix fix pylint issues rename function ...
jgfouca
pushed a commit
that referenced
this pull request
Jun 2, 2017
Fix pylint errors Also, add logging-format-interpolation to list of pylint warnings we don't care about. That way, we can use python3 string formatting without getting warnings. Test suite: code_checker Test baseline: Test namelist changes: Test status: bit for bit Fixes [CIME Github issue #] User interface changes?: None Code review: @JEdwards
jgfouca
added a commit
that referenced
this pull request
Jun 2, 2017
Update to cime5.3-alpha10 Pulling in another 2 weeks of CIME changes into ACME. CIME hash 15297cd from May 2. Required bug fixes for: Changing error to warning for using "-id" with single dash acme templates missing "arg =" Restore pio1 Cmake path pointing to pio2/cmake Update to gcc5.3.0 to avoid internal compiler error Still had 2 fails of acme_developer on penn machine, but look to be weird gcc issues -- var tracking overflow. Same tests passed on redsky, so expecting it will pass on ACME machines. [BFB] * agsalin/update-to-cime5.3.1: (219 commits) Add fix to bug in docn that was in this cime version Fix upstream merge resolution mistake Bug fix: Handle failures to get mem usage from baselines Make single-dash before multichar arg a warning Fix acme template change for new parser Update Sandia worksations to gcc5.3.0 Comment out invalid_args check Revert change in pio1 to point into pio2/cmake Fix single submit Add tests Do not override walltime unless test Update ChangeLog nag compiler needs a width Add checks to verify the create_newcase directory was created as expected removed unpack commented region fix typo in parse_args Bug fixes Minor pylint fix fix pylint issues rename function ...
jgfouca
added a commit
that referenced
this pull request
Feb 27, 2018
Update to cime5.3-alpha10 Pulling in another 2 weeks of CIME changes into ACME. CIME hash 15297cd from May 2. Required bug fixes for: Changing error to warning for using "-id" with single dash acme templates missing "arg =" Restore pio1 Cmake path pointing to pio2/cmake Update to gcc5.3.0 to avoid internal compiler error Still had 2 fails of acme_developer on penn machine, but look to be weird gcc issues -- var tracking overflow. Same tests passed on redsky, so expecting it will pass on ACME machines. [BFB] * agsalin/update-to-cime5.3.1: (219 commits) Add fix to bug in docn that was in this cime version Fix upstream merge resolution mistake Bug fix: Handle failures to get mem usage from baselines Make single-dash before multichar arg a warning Fix acme template change for new parser Update Sandia worksations to gcc5.3.0 Comment out invalid_args check Revert change in pio1 to point into pio2/cmake Fix single submit Add tests Do not override walltime unless test Update ChangeLog nag compiler needs a width Add checks to verify the create_newcase directory was created as expected removed unpack commented region fix typo in parse_args Bug fixes Minor pylint fix fix pylint issues rename function ...
jgfouca
added a commit
that referenced
this pull request
Mar 14, 2018
Update to cime5.3-alpha10 Pulling in another 2 weeks of CIME changes into ACME. CIME hash 15297cd from May 2. Required bug fixes for: Changing error to warning for using "-id" with single dash acme templates missing "arg =" Restore pio1 Cmake path pointing to pio2/cmake Update to gcc5.3.0 to avoid internal compiler error Still had 2 fails of acme_developer on penn machine, but look to be weird gcc issues -- var tracking overflow. Same tests passed on redsky, so expecting it will pass on ACME machines. [BFB] * agsalin/update-to-cime5.3.1: (219 commits) Add fix to bug in docn that was in this cime version Fix upstream merge resolution mistake Bug fix: Handle failures to get mem usage from baselines Make single-dash before multichar arg a warning Fix acme template change for new parser Update Sandia worksations to gcc5.3.0 Comment out invalid_args check Revert change in pio1 to point into pio2/cmake Fix single submit Add tests Do not override walltime unless test Update ChangeLog nag compiler needs a width Add checks to verify the create_newcase directory was created as expected removed unpack commented region fix typo in parse_args Bug fixes Minor pylint fix fix pylint issues rename function ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pulling in another 2 weeks of CIME changes into ACME.
CIME hash 15297cd from May 2.
Required bug fixes for:
Fixes #1343
Fixes #1388
Fixes #1396
Fixes #1401
Fixes #1426
Fixes #1473 (not 100% this one made it in, if not, the next merge will get it)
Still had 2 fails of acme_developer on penn machine, but look to be weird gcc issues -- var tracking overflow. Same tests passed on redsky, so expecting it will pass on ACME machines.